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QUALITY MEASURES FOR SYSTEMS OF ECONOMIC ACCOUNTS 


John Zarb 
Analytical Services 


EXECUTIVE SUMMARY 


This paper reports work in progress on a project called Quality Measures for 


Systems of Economic Accounts (QMSEA). 


The distinctive features of QMSEA are that it — 


iL 


Is designed to review whether present and future systems of economic accounts 
(SEAs) are likely to be “fit for purpose”. 


Looks beyond sample survey data to the full range of input data used to compile 
SEAs. 


Will concord with the IMF Data Quality Assessment Framework (DQAF), an 
international standard being developed, but aims to be more comprehensive 
than DQAF — in both the breadth and depth of auditing quality aspects. 


Will build upon the existing stock of information concerning quality aspects of 
SEAs. 


Is designed primarily for a relatively small, sophisticated group of users (both 
outside and inside the ABS). 


Might in the long run use a combination of ABS self-assessment and external 
assessment (by, say, a consortium of user agencies such as the Treasury, Reserve 
Bank of Australia and Prime Minister and Cabinet). 


May be made available in electronic form so that it can be readily maintained and 
interrogated. 


I think that something like QMSEA is needed because existing quality measurement 


procedures are only partial and give less than comprehensive information for 


assessment of whether SEAs and their statistics are fit for use. 


The QMSEA development strategy involves the following steps — 


Initially defining an ideal set of quality indicators and refining them in 
co-operation with key stakeholders (producers and high-end users of SEAs). 


Conducting a pilot test on a couple of subsystems of economic accounts. 
Existing quality indicators will be assembled, even if imperfect, so that the quality 


indicator matrix can be populated as much as possible, and a program planned 
for generating the remaining quality indicators to populate the matrix. 


After sufficient testing and bedding down on subsystems, QMSEA will be 
promulgated to full systems of economic accounts. 


QUESTIONS FOR MAC 


We would welcome MAC members’ comments on any aspect of the QMSEA 
project, but in particular on the following questions — 


Is the QMSEA project worthwhile, in the light of the audience likely to 
comprehend its outputs? 


Are there aspects of quality for systems of economic accounts not adequately 
addressed in the QMSEA-type approach? What areas may need 
strengthening? 


Are there any suggestions as to how QMSEA should be developed and pilot 
tested? 


Are there any models for ways of presenting, disseminating and browsing the 
suite of quality measures? 


Should QMSEA-type quality audits remain purely ABS self assessments? How 
might the ABS work with external “auditors”? 


This paper represents work in progress and does not necessarily reflect the final 
approach which the ABS will adopt. 
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The role of the Methodology Advisory Committee (MAC) is to review and direct research 
into the collection, estimation, dissemination and analytical methodologies associated 
with ABS statistics. Papers presented to the MAC are often in the early stages of 
development, and therefore do not represent the considered views of the Australian 
Bureau of Statistics or the members of the Committee. Readers interested in the 


subsequent development of a research topic are encouraged to contact either the author 


or the Australian Bureau of Statistics. 
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Analytical Services 


1. OVERVIEW OF QMSEA 


1.1 Purpose of the project 


This project considers the issues that arise when developing a comprehensive suite 
of quality measures for systems of economic accounts (QMSEA). 


The purposes of QMSEA are to help: 


° key users when making informed decisions about the utility of the statistics 
and when assessing the “intelligence” provided by SEAs; 


° producers of SEAs to determine, attain and maintain quality standards. 


1.2 What are “systems of economic accounts”? 


Economic accounts provide a systematic overview of the performance of large parts of 
the economy. They are essentially a set of concepts, classifications and accounting 
rules embedded within a comprehensive and articulated accounting framework. They 
are systems of accounts in the following senses — 


° They are composed of a set of data, usually time series data, that are linked not 
only in an arithmetic compiling sense but also in a socioeconomic accounting 


Sense. 


. They are distinguished from merely a framework of ‘disciplined’ information 
because they employ procedures that transform directly observable items in 
clerical accounts into measures of the economic concepts needed for 
macroeconomic analysis. For instance, in the system of national economic 
accounts, the real world current dollar value of the turnover of goods and 
services is transformed in several complex steps into the key but intangible 
indicator ‘seasonally adjusted chain volume measure of the production of goods 
and services’. 


That is, the system of economic accounts involves a rational accounting framework of 
linked data that are transformed by procedures that involve our best assumptions and 
understandings to produce ‘value added’ measures of concepts for analysis. 


Many of the most important economic indicators are derived from SEAs. These 
indicators play a pivotal role in setting major socioeconomic policies. For example: 
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° changes in Gross Domestic Product (GDP) come from the Australian System of 
National Accounts; 
° Government Financial Net Worth comes from the system of Government 


Finance Statistics; 


° changes in the nation’s Balance of Trade with the rest of the world, comes from 
the system of External Accounts; and 


° changes in various ‘inflation’ indicators (consumer, producer, labour cost and 
international price indexes, as well as price measures derived from the national 
accounts) come from the family or system of price indexes. 


All are considered to be important indicators of economic health. They are keenly 
examined by the key decision makers in both the public and private sectors. 


BOX 1 —- CORE ACCOUNTS MAKING UP THE AUSTRALIAN 
SYSTEM OF NATIONAL ACCOUNTS 


The types of accounts in the Australian System of National Accounts (ASNA) reflect 
major economic processes occurring simultaneously in the economy — namely 
production, the distribution of income, consumption, saving and investment, 
financial flows and asset accumulation. The ASNA is composed of the following 
types of accounts: production accounts; income accounts; capital accounts; 


financial accounts; and balance sheets. 


Each of these accounts is produced for the economy as a whole. In addition the 
income accounts, capital accounts, financial accounts and balance sheets are 
constructed for each of the four domestic institutional sectors. These four sectors 
are the non-financial corporations, financial corporations, households (including 
non-profit institutions serving households), and general government. 


Another group of accounts that is an integral part of the ASNA is the external 
accounts. These accounts record the transactions and financial positions of 
Australia with the rest of the world. In the ASNA, the external accounts cover 
external income, capital, financial and balance sheet accounts. The external 
income account is analogous to the balance of payments current account. Its 
balance — balance on external current account — is the same as, but of opposite sign 
to, the balance on current account recorded in the balance of payments. 


This is an example of one nexus that exists between a number of systems of 
economic accounts considered in this paper. Another example is the state 
accounts, that is, the national accounts dissected into regions: Victoria, New South 
Wales, etcetera. 
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Systems of economic accounts can be very complex and comprehensive. For 
example, consider the national accounts (Box 1). 


The particular accounts discussed in Box 1 — the production, income accounts, 
etcetera — are referred to as the core accounts of the system of national accounts. 
There are, in addition to the core accounts, supplementary accounts referred to as 
‘satellite accounts’ and ‘social accounting matrices’. See Box 2. 


BOX 2 — SUPPLEMENTARY ACCOUNTS 


Recently the core accounts of the systems of national accounts have been extended 
to address the analytical demands made upon them. Various socioeconomic and 
environmental issues are being confronted through supplementary or 
complementary satellite accounts or social accounting matrices. 


For example, consider the release of the Tourism Satellite Account, 1997-98 in 
response to the House of Representatives Committee Report, Taxing Relaxing. 
Three further ABS satellite accounts are foreshadowed for development in 
response to user interest, namely the Household Production, Health, and 
Information Technology and Telecommunications satellite accounts. 


Also, the ABS’s contribution to the development of the System of Economic and 
Environmental Accounts manual, particularly in the field of valuing environmental 


assets within the scope of the System of National Accounts is an example of world 
leading development in this area. The ABS has developed chain volume estimates 
of the economic capital stock of non-produced assets, such as land and subsoil 
assets — items that have probably not been published by any national statistical 
agency before. 


The regular compilation of such comprehensive systems of integrated economic 
accounts, with their various detailed balance sheets, accumulation and flow accounts, 
attempts to aid understanding and analysis of various aspects of an economy’s 
performance. The accounts may also be an aid in assessing the efficacy of economic 
management and of policy intervention. However, besides essentially economic 
considerations, there are important sociopolitical and environmental considerations 
associated with producing SEAs. 


1.3 Justification for the QMSEA project 

Because of the important role information from such SEAs may also have in both 
commercial and financial decision making, as well as in the setting of National and 
State economic policy directions, it is reasonable to expect that the statistical 
information and “intelligence” provided by SEAs can be judged as “fit for purpose”. 
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Lack of a comprehensive set of quality measures for economic accounts 


It appears that the quality of ABS SEAs is generally regarded well by key users. 
However, at present neither the Australian Bureau of Statistics (ABS) nor other 
National Statistical Offices (NSO) have publicly available and agreed sets of 
comprehensive quality measures for such systems of economic accounts. 
Nevertheless ABS is conscious of quality issues and endeavours to provide available 
information about the quality aspects of its products. In ABS there is partial coverage 
of the quality issues in Concepts, Sources and Methods manuals, some Information 
Papers and Explanatory Notes. Where the partial coverage of quality aspects is not 
sufficient for users specific needs, information about quality may be conveyed on 
request to users by either bilateral or multilateral discussions. There is, however, a 
lack of comprehensive information about quality aspects generally available for 
established systems of economic accounts. This situation may be an issue when SEAs 
are the subject of extensive methodological reform and comprehensive information 
about aspects of quality are not available to the user or producer to assess or monitor 
the quality improvements brought about by such changes. 


In recent years, the ASNA has been subject to a number of important changes. These 
have included the switch to a more updated and rational accounting framework, 
introduction of chained price and volume indexes, supply and use balanced integrated 
accounts, as well as some other novel conceptual and methodological changes. About 
the time of these changes three Information Papers (ABS cat. no. 5253.0, 5251.0, and 
5248.0) discussed the changes and provided some information concerning quality 
aspects of the upgraded national accounts. 


The availability of partial quality information and an absence of comprehensive quality 
information for both internal and external assessment purposes when required for 
determining fitness for use is a potential concern. 


External demands regarding data quality 


Acceptance of this lack of comprehensive information about quality aspects is likely to 
decline. Statistical users are becoming more aware of the importance of aspects of 
data quality for various reasons — 


. There is a growing number of users, including Governments (Federal, State and 
Local), paying directly or indirectly for statistical information and expecting 
evidence of “value for money”. 


° With the growing demand for more sophisticated statistical analysis there is 
greater need for comprehensive information about aspects of data quality so that 
the user can decide whether the statistical product is fit for the intended 
analysis. 
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° The socioeconomic consequences of inappropriate policy decisions based on 
data of quality insufficient to the task are becoming greater, as larger populations 
are affected and greater monetary and non-monetary resources are involved. 


Metadata about quality aspects is required by users for informed decision making. 


External demands regarding value for money 


Also, the ABS is under increasing pressure to demonstrate that it has achieved 
improvements in its performance as a government agency, and in particular that it has 
increased both its efficiency and productivity in pursuing its declared Mission of 
assisting with informed decision making. Without appropriate quality measures of the 
key and major statistical outputs of a National Statistical Office, Government and ABS 
management cannot make informed decisions concerning the rational allocation of 
scarce resources to or within the national statistical program. 


For instance, the usefulness of benchmarking indicators for determining changes in a 
NSO’s “efficiency”, “productivity”, “effectiveness”, “efficacy” or “value for money 
performance” for intra-agency and inter-agency comparisons, can be seriously limited 


by the absence of appropriate quality measures of its key statistical outputs. 


Internal demands regarding resource allocation and product improvement 


Also, in times of scarce resources and strongly competing demands, optimal internal 
management and governance of a NSO’s many statistical programs requires that 
quality measures be available for its various SEAs. This information is required so that, 
for example, when a statistical agency is reviewing its work program: 


° it can make improvements to quality aspects in the design of new processes 
associated with a SEA; 


° it can make improvements with regard to quality aspects within an existing SEA; 
and 
° it can assist the users of its SEAs, which can be analytical areas of the NSO itself, 


in making informed assessments as to the utility of the SEA product. 
Comprehensive quality information assists an NSO in the above exercises because: 


° its monetary cost/benefit analyses can include explicit reference to quality 
change, and 


° the effort put into and between SEAs can be balanced according to various 
tensions and demands arising from non-monetary/sociopolitical considerations 
that are dependent on quality features. 
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1.4 QMSEA’s approach to quality measurement 


The QMSEA project takes a holistic approach to SEAs. 


QMSEA’s quality measures will be integrated with other systems of quality 
assessment being developed for systems of accounts and for the data collections 
that flow into SEA compilation. 


Also, QMSEA will integrate with other quality regimes assessing the quality of 
methodologies or procedures employed in SEAs. The methodologies to be 
considered include seasonal adjustment, trending, temporal benchmarking, supply 
and use balancing and price/volume estimation. 


1.5 Who are the clients for QMSEA? 


The QMSEA project aims to articulate a procedure or paradigm for developing 
comprehensive quality measures and indicators for systems of economic accounts, 
which in turn can be assessed by a broad range of interested clients. 


The clients of QMSEA will be both the producers of SEAs and the more 
sophisticated users. 


Producers of SEAs may use QMSEA as an adjunct to the other quality control and 
improvement procedures of their SEAs, their statistical inputs, and their methods of 
transformation and presentation of outputs. 


Although potentially appealing to a broad range of interests, it is expected that in 
practice these QMSEA quality measures will be accessed and assessed by a small group 


of specialist and sophisticated users. 


In Australia this group will include senior advisers to Governments, management of 
the ABS, various agencies such as Treasury, Finance, Prime Minister and Cabinet, 
Reserve Bank of Australia, as well as other economically and statistically sophisticated 
regular users of the data in academia and the private sector. International users may 
be the non-government agencies such as the International Monetary Fund, World 
Bank, Organisation for Economic Co-operation and Development, and the United 
Nations. Also there are the influential financial market rating agencies who presently 
rate not only the performance of economies around the world, but also rate the 
quality of the statistics about the economies. For example, see “Great forgery 
forward”, The Australian Financial Review, 2 April 2001, p. 29. The IMF has also 
reported that some users in the financial markets have already requested of it a form 
of Data Quality Assessment Framework that would permit a country ranking or 
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scoring system for data quality aspects. (See in Section 3.4, “Risks in providing 
summary measures”.) 


The sophisticated users are motivated by a “need to know”, and are relatively better 
placed than others to comprehend the complex multifaceted nature of the QMSEA 
quality information. 


On the other hand, it is thought unlikely that the casual or lay users of the statistical 


results of the systems of economic accounts will have either the motivation to access 
the relatively large bank of these quality measures, or the ability to comprehend their 
meaning without having to expend considerable time and intellectual effort. 


1.6 Breadth of the QMSEA project 


QMSEA recognises that systems of economic accounts are more complex than 
individual data collections, such as the survey of retail turnover. 


Appendix A outlines schematically the features of systems of economic accounts that 
QMSEA is encompassing. 


The structural elements that underlie QMSEA include: 


© six streams of input data used to compile systems of economic accounts; 
° twelve facets of quality, each with various dimensions; and 
° three phases of SEA compilation. 


Six streams of input data to SEAs 


Systems of economic accounts draw upon a number of input data streams besides 
probability based sample surveys conducted directly by the ABS. The other data 
streams are: 


° non-probability based sample surveys; 

° censuses or complete enumerations; 

° administrative or transactional by-product collections; 

° market place and media intelligence; and 

° the synthesis or “modelling” of some or all of the previous data streams. 


The six streams of input data also come from a number of broad sources, some 
beyond the direct control or responsibility of the ABS. The three broad sources are 
the ABS, the rest of Australia (ROA), and the rest of the world (RoW). 


Data sources other than the ABS may raise particular challenges for assessment of 
quality or standards, and/or the publication of those assessments by the national 
statistical office. This arises in large part because quality indicators for these sets of 
input data are rarely available from their source. ABS experience has shown that with 
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administration by-product collections, for instance, quality issues regularly arose that 
needed to be managed carefully to ensure the continuing release by ABS of high 
quality time critical SEA data. 


The ABS recognises that it is important not to view administration by product 
collections as a simple data processing exercise. The major concerns relate to 
consistent and appropriate concepts and definitions, timeliness and care in 


compilation. 


Difficulties of maintaining quality assurance with externally sourced input data can 
arise because of various factors, such as: 


° changes to legislation. For example, a key objective of the revised Customs 
legislation introduced into Parliament late 2000 was to institute tighter 
arrangements for the release of trade data relevant to the external accounts; 


° changes to government or agency policy. For example, the recent policy 
changes concerning the lodgement and payments associated with the Business 
Activity Statements have meant delays and gaps in data, though the ABS does 
not presently use such data to compile its SEAs; and 


° changes in Information Technology. These changes can cause important 
operational problems in transferring, interrogating and editing data in a timely 
manner. 


There are also issues regarding whether the degree of data sharing between various 
agencies is appropriate and respectful of confidentiality considerations. There have 
been concerns recently about whether there is a perception of administration 
by-product data sharing between Australian Prudential Regulation Authority (APRA), 
Treasury, Reserve Bank of Australia (RBA) and ABS. Federal legislation concerned 
with these issues is presently being drafted and may have some bearing on quality 
assessment issues for SEAs. 


External data sources evidently raise many considerations for quality assurance and 
measurement, which sometimes need to be addressed by the ABS and the external 
source entering into explicit arrangements which include: roles and responsibilities of 
the respective agencies, agreement on questionnaire content and design, timely data 
access and transference procedures, editing and querying protocols, including 
whether the ABS is allowed to query respondents directly, and appropriate data 
feedback loops to the source agency. 
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Twelve facets of SEA quality 


QMSEA identifies and discuss issues in connection with about twelve aspects or facets 
of quality associated with systems of economic accounts. They are: 


: accessibility 


: accuracy 

° compliance 
: consistency 
. continuity 


° frequency 
. longevity 

° lucidity 

° relevance 
° revisability 
° sense 

° timeliness. 


The Glossary contains more detailed descriptions of these twelve facets. 


The facets of quality are often interrelated and overlapping to some extent. In 
practice, tradeoffs and compromises need to be considered when striving to enhance 
one or more of these aspects of quality for a particular data item. 


For instance, there may be tradeoffs between: timeliness and accuracy; accuracy and 
revisability; relevance and longevity; relevance and accuracy; sense for large domains 
and relevance for subdomains. 


The existence of such tradeoffs makes any global or composite quality indicator for a 


SEA very problematic. 


Three phases of SEA compilation 


These facets of quality are meant to be appropriate, in varying degrees, to the 
assessment of the three phases that data flow through during the compilation of SEAs. 
The three phases are: 


° the input data 
° the transformations of the input data 
° the transformed output data and the “intelligence” generated. 


Within each of the twelve (or so) quality facets there are various dimensions 
depending on the particular phase of SEA compilation being considered. 
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First phase 


Some issues involving the assessment of the quality of the input data streams were 
discussed briefly above. 


Second phase 


In addition to the input data phase, a quality assessment procedure for SEAs is 
required to assess the quality of the second phase, which involves the various 
transformation methodologies that each SEA employs. This is essential as the SEA 
estimates are sometimes sensitive to the assumptions and nature of the 
methodologies employed. 


For example, estimates of capital services and multifactor productivity are very 
sensitive to the assumptions that underlie the perpetual inventory method used to 
estimate the capital services. 


Similarly, the valuation of employee held shares and share options as a form of 
remuneration is problematic and dependent upon assumptions; source data is 
difficult to obtain and because of this an issue is whether the valuation should be 
imputed for each reference period or valued at the time of realisation. 


Also, the allowance for quality changes in a variety of new products and services is 
sensitive to the assumptions made about the "worth" of the quality changes. 
Improvements to quality can be in areas such as safety, durability, comfort and 
energy/resource efficiency, and are difficult to estimate as there is a degree of 
subjective judgement involved. 


The method of temporal indicator-benchmarking that is used to generate quarterly 
series from sets of indicator series in conjunction with old annual data can have 
important consequence. The quality of the results depend on the level of 
disaggregation at which the methodology is applied. (Cainelli and Lupi, 1999). 
Different price and volume estimation methodologies can also have significant 
impacts on the key estimates of SEAs as can the seasonal adjustment and trending 
methodologies employed. All of these types of transforming methodologies used by 
SEAs are to be evaluated by QMSEA. 


Third phase 


Unlike a particular data collection quality assessment, a SEA quality assessment has a 
diverse range of outputs to evaluate in the third phase. Systems of economic accounts 
invariably produce statistics in seasonally adjusted and trend from as well as original 
form. These forms of the data relate to flow and stock variables, as well as 
combinations of these distinctly different variables. For example, GDP per capita is a 
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composite indicator constructed of a flow variable in its numerator (GDP) and a stock 
measure in its denominator (population at a point in time). Consequently variables of 
this nature, referred to here as “flocks”, can have quality attributes different to their 
antecedents. All these variables also cover a variety of economic attributes, such as 
current monetary values, chained volumes, physical quantities, price indexes and 
implicit price deflators. 


A wide range of variable types 


It is also important to note that these SEA data are expressed as one or more variables 
in the form of levels, changes in levels, percentage changes, and ratios of various 
variables. This diversity of output variables from SEAs adds an additional degree of 
assessment responsibility to QMSEA not generally borne by other particular data 
collection quality regimes. This aspect is important to note, because although quality 
measures from a particular data collection process may appear acceptable to a user for 
a particular data item (usually in raw form), their eventual transformation through a 
SEA into, say, derived seasonally adjusted and chained volumes, etcetera, may alter 
materially their “fitness for use”. 


Also, the aspects of quality may vary between related forms of a variable. For example, 
a quarterly percentage change of a particular data item may be assessed as having such 
and such a degree of accuracy, whereas its related level form could be less accurate 
because the errors in the level estimates are known to be appreciable but of a fairly 
constant proportional nature. In this example the proportional errors of the level 
estimates cancel out in the percentage movement estimates. Similar considerations 
apply to the other variable types. 


QMSEA is to address these types of quality considerations because users’ needs vary 
between SEA types and various analysis applications. For example, in the annual 
national accounts it can be the current dollar level estimates per annum that are of 
interest, whereas in the quarterly national accounts it is the quarterly percentage 
movements of the chained volume seasonally adjusted data item that is of primary 


concern. 


Beyond sampling errors 


In developing a comprehensive suite of quality measures for a system of economic 
accounts, it is recognised that “sample survey standard error” is associated with only 
one of the input data streams to systems of economic accounts; the probability 
sample survey based results. Non-probability based or purposive surveys such as used 
for the Consumer Price Indexes (CPD or Producer Price Indexes (PPI), which feed into 
the system of national accounts, do not have associated with their results a 
quantifiable sampling standard error. Nor do the other input data streams. For 
quality assessments of systems of economic accounts the role of sampling standard 
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error is a relatively minor one compared with that of non-sample errors. This is 
because there is greater potential for non-sample errors to be more prominent factors 
in SEAs than sampling standard errors. Further, the sampling standard errors’ 
importance in the exercise of informing the user about the overall accuracy of a 
system of economic accounts is expected to decline in importance in the future. See 
Box 3. 


BOX 3 — THE NEED TO LOOK BEYOND SAMPLING ERRORS 


The relative decline in importance of “sampling standard errors” is expected for a 
number of reasons. 


Firstly, issues of provider load, emerging opportunities and continuing tight 
budgets will mean less reliance on sample surveys by economic statisticians and a 


growing dependence upon administrative by-product or transactional data sources. 
The increasing use and reliance on administrative by-product and transactional 
data, like the Australian Tax Office’s Business Activity Statements (BAS), EFTPOS 
and “scanner” data, is expected to increasingly replace various ABS conducted 
probability based sample surveys. These input data sources/sets will inevitably be 
imperfect from a conceptual and compilation perspective, and will provide difficult 
challenges because the e-commerce based transactions datasets will be huge. 
(Edwards, 2001, paras. 11 and 31) Consequently, the source and significance of 
sample standard errors in a suite of quality measures for SEAs is expected to 
decline in a relative sense. 


Secondly, there is an increasing realisation and awareness that the “sampling 
standard error” is only one of a number of quality aspects of a probability based 


sample survey estimate, and that it may not be the most important aspect simply 
because it may be more easily estimated or “given a number” than the other quality 
indicators associated with the a probability based sample survey results. Simply 
put, users are becoming aware that the sampling standard error is not a guide to 
the total unknowable degree of accuracy. 


Thirdly, with increasingly complex compilations of various input data streams in 
SEAs, and transformations into constant price or volume measures and seasonally 
adjusted or trend aggregates, sampling standard errors for such statistics become 
either irrelevant, problematic, or too costly to compute for systems of economic 
accounts. 
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1.7 QMSEA’s distinctive focus 


ie 


QMSEA focuses on: 


designing for a relatively well developed and mature economy a 
comprehensive set of quality measures for assessment of economic accounts 


produced by a statistically sophisticated national statistical agency, such as the 
ABS. QMSEA will not be probability sample survey-centric; 


a system for monitoring developments over time of such quality measures 
within a single country, rather than for inter-country comparisons. In this 
respect, the project will complement the International Monetary Fund’s 
various quality projects covered under the auspices of the Data Quality 
Assessment Framework. The DQAF is discussed in some detail later; 


the Australian System of National Accounts (ASNA) as its primary case study, 
but should be applicable to other systems of economic accounts, such as the 
Balance of Payments and Government Finance Statistics; and 


aspects of economic accounts’ quality that should be of interest to the more 
sophisticated users, such as the Government, Treasury, Reserve Bank of 
Australia, other international NGOs and similar NSOs. 


1.8 Outputs from the QMSEA project 

Because of the large quantity and complex nature of the information concerning 
quality indicators, it is envisaged that the QMSEA assessment information would be 
banked, archived and distributed electronically. See Box 4. 


To support a free ranging interrogation procedure of the type described in Box 4, 


large banks of quality information may need to be collected and arrayed. Such an 


assembly of information is here called the quality matrix. A potential concern is 


whether it is feasible and necessary to fully populate the quality matrix. There is 


concern about the time, effort and opportunity cost, and resources required to be 


borne in an attempt to populate the quality matrix. This question is not easy to 
answer without the insight from trials. Also, the answer will be partly determined by 
what is considered to be the primary aim of QMSEA. This issue is discussed later in 
Section 3.3. 
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BOX 4 — ELECTRONIC DISSEMINATION OF QMSEA’S QUALITY MEASURES 


Electronic access to QMSEA information should permit the enquirer to trace the 
quality audit trail in many notional directions. For instance, the enquirer should be 
able to: 


° search vertically, in the sense of “drilling down” to more detailed levels of a 
particular aggregate’s component information; 


° “pan horizontally” in the sense of tracing the quality audit trail through 
various components of the indicator at a particular level of aggregation; 


° follow either of these views over consecutive reference periods, or fora 
particular reference period over time as it is re-estimated. 


To appreciate how this quality search might be performed consider the 
fundamental supply and use relationship of the system of national accounts. It can 
be expressed as: 


Ore Mi — Cir Oi ahha selina Vase 


where supply is composed of output (O), taxes less subsidies (T), imports of goods 
and services (M) respectively and equals use. Use is in the form of intermediate 
consumption (Ci), final consumption (Cf), gross fixed capital expenditure (GFCE), 
changes in inventories (cIn), acquisition less disposal of valuables (V) and exports 
of goods and services (X) respectively. 


The user should be able to trace the quality assessment of anyone of the 
components in the above identity, or may select an item like output and investigate 
its components by considering dissections such as industry, sectors, or products. 
Such an enquiry may additionally select a particular reference period and monitor 
how the quality measures have varied over time. Alternatively, the user may select 
a string of reference periods and observe how the quality measures have changed 
over this analysis span. 


Such quality assessment information might also be made available internationally 
via the presence of a hyperlink, allowing users to move directly from, say, the IMF 
bulletin board to the ABS website(s). Such types of arrangements, however, are 
not free form organisational complications or difficulties. 
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2. QMSEA’S RELATIONSHIP TO OTHER QUALITY 
MEASUREMENT SYSTEMS 


QOMSEA differs in some important respects. QMSEA aims to: 


concentrate on standard errors; 


flowing into SEA compilation, 


for casual or lay users; 


of transformations; and 


° be applicable to the broad variety of data output from SEAs and not just a 
single data form. 


In relation to other quality assessment regimes in the ABS, other NSOs and NGOs, 


° be more comprehensive and detailed in its quality assessment, and will not 


° not be sample survey-centric — it will also consider the five other data streams 


° have broader application than just data collections by considering the effects 


° be suitable only for regular sophisticated users and producers of SEAs and not 


2.1 Development of international quality standards 


The search for universally agreed and accepted quality statistics has been frustrating. 


De Vries (1999) recently observed: 


“Another example is the Mahalanobis Committee, created by the International Statistics 
Institute in 1995. Its aim was to develop ‘statistics about statistics’. There have been no 
results so far, the reason being on one hand a total lack of interest in participating in the 
committee’s work, and on the other hand widespread disagreement about how to tackle 
the issue.” 


De Vries concluded: 


“After all, I think there is some irony in the fact that while statisticians are constantly 
trying to agree in great detail which internationally comparable information (in terms of 
definitions, classifications, other methodologies etcetera) to ask from business, 
institutions and households, they do not wish to agree on measurement of their own 
operations, and are apparently unable — up to now — to agree on universally accepted 
definitions of concepts such as non-response or indeed statistician.” 


Over the last decade there has been growing interest amongst NSOs in developing 
various systems for measuring or attempting to assess the quality of either particular 


data collections or systems of economic accounts. An ABS “quality” policy was 


originally approved by the ABS Division Heads Meeting in February 1988. (ABS, 1992) 
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In this context, the term “data collection” refers to a particular subset of 
macro-economic data, such as capital expenditure, retail turnover or labour force 
Statistics. 


The term “system of economic accounts” refers to large sets of integrated accounts, 
such as the national accounts or balance of payments, etcetera. 


Some of the recent attempts at developing quality assessment systems strive to be 
more systematic and comprehensive than the earlier attempts. And some have been 
very ambitious. See Box 5. The IMF’s present project concerning quality systems, the 
Data Quality Assessment Framework (DQAF) is one of the latter type, and is discussed 
briefly later. 


BOX 5 — EVOLUTION OF THE INTERNATIONAL QUALITY FRAMEWORKS 


It is not possible to nominate a definitive date or agency as being the starting point 
or instigator of the present movement for quality information. However, one 
notable example of an agency that spelt out broad quality considerations is the 
Office of Budget Management, USA, in its Statistical Policy Directive No. 3: 
“Compilation and Release of Principal Federal Economic Indicators”. The directive, 
issued 25 September 1985, gave guidance to federal USA agencies on the 
compilation and release arrangements for economic indicators. No previous 
watershed has so influenced many quality features of a nation’s statistical agencies 
and their systems of economic accounts. The directive’s intent was: 


“., to ensure that the Federal data and estimates used to assess current economic 
conditions meet high standards of reliability and usefulness and that agencies release 


them in a fair and orderly manner.” (Waite, 1995, p. 3) 


In addition to adding a performance evaluation every three years, the directive also 
included stringent procedures for announcing changes to data collection, analysis 
and estimation methods. 


In the early 1990s there were various attempts by NSOs to assess the quality of key 
national account aggregates by studying their revision history. ABS has 
subsequently published a number of revision history analyses. (ABS, 1996; 
Rossiter, 1996; Webster, 1998) Such study can shed light on whether there are 
serious biases in the results, and whether source data or methodologies are 
contributing more or less to the estimates revisability. While shedding some light 
on one aspects of quality, this approach was recognised as addressing only part of a 
broader key question of whether the data are reliable, or fit for use. It was 
acknowledged that the absence of revision does not necessarily mean that the data 
are accurate, because a wrong or less than accurate number could remain 


unrevised. Also, in some instances the ability to revise data is constrained by 
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various factors. For instance, as a result of the way the CPI is compiled, and 
because of the manner it is used for inflation indexation in contractual and social 
security type arrangements, revisions to the initial CPI estimates do not occur. This 
situation has prevailed even though aspects of the index methodology employed 
for the CPI were known to be biased. 


Associated with the interest in revisions is an interest in accuracy. Accuracy is 
concerned with the proximity of an estimate to the ideal or true but unknown value 
of that component. There is a natural or instinctive desire to know how accurate 
the data are, but unfortunately the nature of systems of economic accounts, like 
the national accounts, balance of payments and government finance statistics, 
makes it extremely difficult to ever have a tangible benchmark against which to 
measure accuracy. Nor is there, for these accounting systems, a statistical theory to 
appeal to so as to produce a quantitative measure such as “sampling standard 
error” for the key outputs of such systems. It is important to note that the true 
value is never known, and that in practice there cannot be an overall measure of 
accuracy or total error. Consequently, accuracy of SEAs is in practice evaluated by 
considering the potential sources of error and considering whether they have been 
minimised optimally. 


In 1997, Statistics Canada considered a strategy for developing quality assessments 
statistics for the Provincial Income, Production and Expenditure System of 
accounts (PIPES). It was observed that: 


“Meeting this goal will be one of the major objectives of Statistics Canada over the next 
few years. Unfortunately, the current absence of quantifiable measures of accuracy makes 
it difficult for the bureau to decide on which components are the most critical for 
improving the reliability of the provincial accounts. We desperately need yardsticks to 
help us measure reliability and quality.” (Statistics Canada, Unpublished internal memo, 
April 1997) 


And, the desire to have a measure of accuracy is still strong in some agencies. This 
is indicated by the European Court of Auditors recently seeking international 
assistance in developing error margins or measures of accuracy for key national 
account aggregates for the 15 Member State economies in the European Union. 
The Court of Auditors describes itself as the “financial conscience” of the Union, 
and needs to monitor for fraudulent practices in the preparation and presentation 
of the systems of economic accounts of member countries, and in the stewardship 
of the allocation of financial assistance to member countries based on their 
reported national accounts. However, no statistical methodology is presently 
available to deliver that result concerning accuracy. 


In the mid 1990s, the International Monetary Fund reacted to a number of 
economic/financial crisis and instances of serious misreporting that threatened its 
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effectiveness and credibility. Fearing that its ability to meet its responsibilities for 
surveillance of members’ economic policies and management of the IMF’s loan 
program was being adversely affected by poor quality data, it started to develop a 
quality reporting system that looked more broadly than just aspects revisability and 
accuracy. An overview of this work is found in “Towards a Framework for Assessing 
Data Quality” (IMF/Carson, 2000). Their present work complements the quality 
aspects of the IMF’s earlier Special Data Dissemination Standards (SDDS) and 
General Data Dissemination Standards (GDDS). In recent years this quality 
assessment work has increased in intensity, with the IMF adopting a development 
process for the quality framework which has been consultative with various NSOs 
and iterative in its evolution. Their work in this area is far from finished, though it 
is relatively well developed compared to that of various individual NSOs. 


In addition to the IMF’s first specific data quality framework drafted for the national 
accounts last year June, 2000), other specific frameworks have been drafted 
recently for the balance of payments, the analytical accounts of the central bank, 
the producer price index, and government finance statistics. Progress in 
developing such frameworks as DQAF or QMSEA is, however, slow and arduous 
because construction of a coherent and comprehensive system is difficult in a field 
that is conceptually and practically difficult, very large and complex. 


2.2 Role of international standards 


A feature of the IMF quality framework DQAF is that there is a relatively close mapping 
to some existing international statistical standards, concepts and manuals. For 
example, the 1993 System of National Accounts and the fifth edition of the IMF’s 
Balance of Payments Manual are used as benchmarks for definitions, concepts, 
classifications and valuation methodologies. ABS generally complies with these 
standards and so the IMF quality framework provides a basis or foundation on which 
to build a more comprehensive quality assessment system for the ABS’s economic 
accounts. 


There are sound reasons for generally complying closely with international standards, 
which have positive impacts on aspects of quality. For instance, the statistics from 
different countries are more comparable. This may be very important from a 
relevance perspective, particularly for those users interested in comparing Australia’s 
economic performance with other countries. This consideration may be so important 
as to outweigh the qualitative disadvantages of following an international standard. 
Also, it may be possible to have a SEA with a “dual” basis, thereby providing users with 
different views of economic activity, and satisfying different needs. 
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BOX 6 — THE SNA 1993 STANDARD FOR NATIONAL ACCOUNTS 


SNA 1993 (which was implemented by the ABS in late 1998) was very different from 
its predecessor, SNA 1968. 


It was changes in the structure and nature of the economy, the increasing 
sophistication and growth of financial markets and their instruments, emphasis on 
the interaction of the economy with the environment and other considerations, 
such as a need to clarify and simplify the prevailing accounting system, that 
indicated a need to update the SNA. 


The task of modernising SNA commenced in the mid 1980s, and the new version 
was released under the auspices of the Commission of the European Communities 
(now Eurostat), the International Monetary Fund, the Organisation for 
Co-operation and Development, the United Nations and the World Bank in 1993. 
Until the implementation by ABS of SNA 1993 in 1998, the system used through the 
1990s was based on the conventions and practices prior to 1968 and evidently not 
up to date. 


Recently there has been much talk about the so-called “new economy”. 


“The term New Economy refers to a set of qualitative and quantitative changes that, in the 
last 15 years, have transformed the structure, functioning and rules of the economy. The 
New Economy is a knowledge and idea based economy where the keys to job creation 
and higher standards of living are innovative ideas and technology embedded in services 
and manufactured products. It isan economy where risk and uncertainty, and constant 


change are the rule rather than the exception.” (Edwards, 2001, para. 22) 


Questions may therefore be raised about how well SNA 1993 provides a framework 
for assessing the macro-economic consequences of the so-called new economy. 


In the mid 1990s Waite said: 


“More seriously, I believe that by most criteria US economic indicators meet or exceed 
reasonable criteria for timeliness and also meet those criteria for reliably measuring 
short-term economic change. This, however, is quite different from saying that our 
statistics are keeping pace with longer run forces shaping the US, indeed the world, 
economy. Our economies are changing faster than our ability to measure them. 
Economic output is increasingly conceptual rather than physical. The form of output and 
the means of production have been increasingly less physical and tangible, hence more 
difficult to measure. ... This trend is sure to continue into the 21st century.” (Waite, 
1995, p. 10) 


In Australia, as elsewhere, Waite’s observations are pertinent; commodities and 
industries grow at different rates, and new products come onto the market and old 
ones leave on a regular basis. It is therefore important that our SEAs are flexible 
enough to keep pace with the change. 
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Australia’s implementation of SNA 1993 reflects local conditions and requirements, 
with isolated instances of departure from the standards because of strong user 
preference for an alternative view. 


“The departures are relatively minor and, consequently, they do not affect the 
comparability of national account information reported by the ABS to international 
organisations such as the UN and the OECD to a significant extent.” (ABS cat.no. 5216.0, 
2000, p. 8) 


Refer to Box 7 also. 


However, there are some concerns associated with attempting to base a quality 
assessment system solely around existing international standards. International 
standards are not necessarily monolithic, homogeneous, consistent, generally 
complied with, integrable, or best current practice, as discussed briefly below. 


Internationally accepted statistical standards can be useful, but they are not always the 
currently best available practice. For instance, consider SNA 1993, which is the 
present standard adopted by the ABS for its national accounts. See Box 6. 


Unlike DQAF, one aim of QMSEA will be to explicitly assess and review whether the 
SEA methodologies and basis are relevant or appropriate for the present and 
foreseeable future circumstances of Australia. The QMSEA philosophy is that the 
quality of a system of economic accounts should be measured and judged with 
reference to the cognitive purpose of the system of economic accounts, not solely 
against some generic international standard about which a system of economic 
accounts might be based. QMSEA will not automatically assume an international 
standard is necessarily the best or appropriate benchmark for the above reasons and 
those discussed in Appendix D. 


Although not hidebound by generic international standards just because they exist, 
QMSEA: 


° will concord with DQAF where possible; 
° will extend DQAF for statistically sophisticated countries and users; and 
. may be used by the ABS to influence international standards. 


While many countries of the world have expressed their intentions to produce their 
statistics in an SNA 1993 framework and format, this by no means guarantees that their 
data conform to the SNA 1993 standards. Implementation by NSOs of international 
standards like the SNA 1993 tends to be selective and judgmental. 


“Without some assessment of the quality of the data, which is usually difficult in practice, 
the mere presentation of the data in the right format may be no more than 'window 


dressing’ to satisfy the demands of the agencies concerned.” (Hill, 1999, p. 1) 
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As indicated earlier the ABS has undertaken to implement SNA 1993 as far as 
practicable. Australia has probably gone further than almost any other country toward 
implementing the full standard. See Box 7. 


BOX 7 — ABS COMPLIANCE WITH THE SNA 1993 STANDARD 


At the request of the ABS, Professor Peter Hill (the principal author of the SNA 1993 
manual) evaluated the ABS implementation of the standard. His conclusions are 
summarised below. 


ABS produces both regular releases of annual national accounts, annual state 
accounts and quarterly national accounts on an SNA 1993 basis. With regard to the 
annual national accounts Prof. Hill has stated: 


“The set of annual accounts published by the ABS in April 1999 seem to be one of the 
most complete sets of accounts on a 1993 SNA basis yet available in the world. The more 
or less complete set of sectoral accounts consisting of income, capital and financial 
accounts plus balance sheets for all sectors of the economy (except NPIs serving 
households) represents a comprehensive implementation of the SNA which is remarkable 
for the inclusion of sector balance sheets which very few countries have been able to 
compile up to now. The accounts also incorporate the new supplementary accounts 
showing ‘adjusted’ disposable income and ‘actual consumption’ for households and 
government, concepts introduced for the first time in the 1993 SNA to which considerable 
importance was attached in the revision of the SNA.” (Hill, 1999, p. 9) 


These annual national accounts also incorporated data relating to the capital stock 
and multifactor productivity measures, which previously had been published 
separately. In addition, this comprehensive system of national accounts was also 
fully consistent with the supply and use and input-output tables that ABS published 
the month before. This closer integration of both the input-output data and capital 
stock data (the latter albeit experimental at this stage) with the compilation of the 
main national accounts was strongly recommended in SNA 1993. This is 
recommended because it tends to improve the quality of all the constituent 
datasets within the SEA by confronting and potentially resolving inconsistencies 
which are indicative of errors in one or more of the component subsets of data. 


2.3 Leadership in setting standards 

The favourable, informed and authoritative assessment of ABS’s implementation of 
SNA 1993 (Box 7), has helped establish ABS credentials for promoting world best 
practice for adoption in international standards and the improvement of SEA quality. 
These ABS improvements are not inconsequential for any consideration of SEA 
quality. See Box 8. 
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BOX 8 — ABS LEADERSHIP IN ECONOMIC ACCOUNTING 


Examples in regard to the national accounts are the treatment of the output of 
financial institutions services indirectly measured (FISIM), the use of annually 
re-weighted chain indexes in preference to less frequently re-weighted indexes for 
the measurement of price and volume changes, and the upgraded estimation of the 
stock of capital and related statistics. 


As to capital stock estimates, the ABS approach figures prominently in a 
recently published OECD manual concerning capital stock and productivity, 
and is given as an example of a conceptually superior method. This 
methodological improvement should strengthen the links between the flow 
accounts and balance sheets of the SNA. That improvement should also assist 
with better estimates of the consumption of fixed capital and thereby better 
estimates of Net Domestic Product (NDP). The provision of good quality 
estimates of NDP and GDP for the economy may be a very significant 
advancement for analytical purposes and policy intervention. In this regard 
QMSEA provides a medium through which the ABS can gauge weaknesses in 
methodologies in ASNA/SNA 1993 and other SEAs, and thereby help indicate 
areas where improvements have been or should be made. 


Some other areas for further quality assessment development and leadership by 
the ABS are chain linked price and volume methodologies, revision analyses, and 
the seasonal adjustment and trend estimation methodologies applied to the key 
quarterly aggregates of the systems of accounts. 


As to trend estimates, Australia is one of the few NSOs (probably the only one) 
to give prominence to quarterly trend estimates from SEAs for the purpose of 
analysis. For example, in the Explanatory Notes of its National Income, 
Expenditure and Product publication ABS states: 


“Given the qualifications regarding the accuracy and reliability of the quarterly 
national accounts, the ABS considers that trend estimates provide the best guide to 
the underlying movements, and are more suitable than either the seasonally 
adjusted or original data for most business decisions and policy advice.” 


ABS has a preference for influencing international standards, general statistical 
standards, economic accounting standards, and quality standards, rather than 
launching off on completely disjoint efforts. This is shown by the participation of ABS 
in various international forums. ABS has shown statistical leadership in a number of 
important areas and has had a good deal of influence on System of National Accounts, 
Balance of Payments, System of Economic and Environmental Accounts, prices 
manuals, etcetera. For further details refer to Appendix D — QMSEA and International 
Standards. 
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2.4 QMSEA and the IMF’s Data Quality Assessment Framework (DQAF) 


The IMF’s work on quality frameworks has the strong support of the ABS. This is 
because the ABS regards quality assessments of its statistics and methods as extremely 
important, both to determine areas that are deficient so that improvements can be 
made, and to assist users in their understanding of the statistics and with the more 
effective use of them. The ABS Corporate Plan 2000 states: “We are also open about 
the quality of statistics, so that users can better understand and interpret them.” 


ABS also supports the DQAF because of the transparency it may provide to other 
countries SEAs and statistics. 


It is expected that QMSEA will draw upon the IMF framework where appropriate, 
modifying and generally extending and deepening the aspects of quality assessed in 
each of the systems of economic accounts, to suit local conditions. This approach to 
the IMF’s DQAF is feasible because their assessment frameworks presently give room 
for flexibility to take account of the various individual country's circumstances. A 
prescriptive “one size fits all approach” was discouraged by the IMF for various 
technical and non-technical reasons. 
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3. IMPLEMENTING QMSEA 


As discussed above, systems of economic accounts like the Australian System of 
National Accounts can be very large, complex and comprehensive sets of time series 
data, which are the result of various sophisticated transformation methodologies. 
Developing and implementing a quality assessment procedure for such SEAs is not 
therefore a simple or straightforward task. Given this situation with SEAs it is 
intended to trial QMSEA on subsystems. The following strategies for implementing 
QMSEA are being considered. 


3.1 Agreeing on quality criteria 

A suite of quality indicators will be defined and described for the three phases that 
data pass through in an SEA compilation. This proposed suite of indicators will then 
be discussed with a selection of the target audience. Here it should be remembered 
that the QMSEA audience is considered to be a relatively small group of statistically 
sophisticated and aware producers and regular users of SEAs. The aim is to agree 
upon a suite of quality indicators that are comprehensible and comprehensive for the 
task of assessing quality in the many diverse situations SEAs are to be used. Preferably, 
the quality indicators should not be ambiguous, and as much as possible they should 
be orthogonal to each other, in the sense that they individually capture one feature of 
quality not represented to any significant extent by another indicator. 


For example, including in the suite a quality indicator referred to as “reliability” 
may seem intuitively sensible, but in practice the term “reliable” is ambiguous. 
The term “reliable” is thought by some people to refer to aspects of “accuracy”, 
while for others it is thought to relate to aspects of data “revisability”. The ABS 
now prefers the term “revisability” to “reliability” in its discussion of quality 
aspects. The term “reliability” in the QMSEA context means neither of the above 
indicators, and takes on a global composite meaning. Here it refers to the 
degree that an estimate or piece of SEA information rates against all of the facets 
of quality, not just the single facet of, say, accuracy. That is, an estimate is said to 
be “more reliable” if it rates more highly against the suite of QMSEA criteria than 
another estimate. 


3.2 Testing QMSEA on subsystems of the economic accounts 
Before attempting to implement QMSEA on the gamut of economic accounts, we 


would wish to undertake some small—medium scale “proof of concept” studies. 


There is a need for the exercise to reasonably challenge the features of the procedure 
without, however, overwhelming its development with too much complexity or 
sophistication. The case studies should also be in areas where interest in the results is 
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likely to be more than just academic. And sufficient quality indicator information 
should be relatively easy to obtain or generate during the “proof of concept” stage. 


The large ASNA may be broken into various smaller segments for testing. For example, 


one of the specific core accounts might be selected. Alternatively, a principal aggregate 


might become a specimen for investigation. Besides parts of the core accounts there 


are the satellite and social matrix accounts to consider. Box 9 suggests some segments 


of the system of national accounts that may be fruitful candidates to trial QMSEA on. 


BOX 9 — POSSIBLE TESTBEDS FOR THE QMSEA CONCEPT 


The recently upgraded capital stock and multifactor productivity estimates are an 
important improvement of the ASNA/SNA 1993. These estimates have been the 
subject of a feature article that has said: 


“The ABS is firmly of the view that the upgraded capital stock and MFP estimates are 
significantly better than those previously published. Nonetheless, the estimates, 
particularly those for capital services and MFP, are very sensitive to the assumptions that 
underlie them. For this reason, it has been decided to label the new estimates of capital 
services and MFP as ‘experimental’, pending a user review of the methodologies.” (ASNA 
1997-1998 , ABS cat. no. 5204.0, p. 8) 


Because of the importance of these estimates for economic analysis, their 
experimental nature, and pending review, this subdomain of the ASNA may be an 
ideal set of data on which to pilot test QMSEA. 


balance of payments and international investment position aggregates. Selecting, 
say, one aggregate from each of the four subjective rating categories (good, fair, 
poor, very poor) and applying the QMSEA quality criteria may prove to be inform- 
ative and helpful in reducing the apparent degree of subjectivity in the present 


be made carefully to ensure that the task is not overwhelmed by the complexity of 
following the quality audit trail back through too many data transformations and 
amalgamations of different input data streams. An example set could be the 
chained volume estimates for Household Final Consumption Expenditure (rated 
Good), Government Final Consumption Expenditure (rated Poor), Public 
authorities Changes in Inventories (rated Fair), Private non-farm Changes in 
Inventories (rated Poor), and Farm Changes in Inventories (rated Very Poor). 


In principle, one of the core accounts of the annual national accounts, such as the 
production account might be considered. However, as in the case of selecting too 
high an order aggregate, this exercise may be overwhelmed by the complexity of 
following the quality audit trail back through too many data transformations and 


amalgamations of different input data streams. 
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With regard to satellite accounts the ABS has developed and published only one 
such system, the Tourism Satellite Account 1997-1998 (TSA). Because its 
development has occurred recently qualitative and anecdotal information about 
aspects of quality may still be fresh in the memories of the producers, and 
consequently using this subsystem as a case study may be a valuable means for 
recording this information before it is lost from corporate memory. Presently the 
TSA publication does not discuss aspects of quality for this dataset in any detailed 
or specific sense; about one page deals with general aspects of the TSA’s quality. 
Another potential advantage of applying QMSEA to TSA is that there is interest in 
repeating the exercise for another reference period, and a quality assessment may 
contribute to improvements in the next round of estimates, which are likely to be 
part user funded. Because of the “youthfulness” of the TSA 1997-1998 and the fact 
that its dataset relates to only one reference period (1997-1998) some of the 
QMSEA guality criteria are not pertinent, such as revisability, longevity, etcetera. 
However, as a matter of course it is thought sensible to apply QMSEA during the 
development stages of any of the next sets of satellite accounts. Applying a quality 
procedure formally during the development stage will involve some effort over and 
above that which is expended anyway when informal assessments about various 
aspects of quality are made in initially setting up these satellite accounts. However, 
proceeding formally with quality assessment during development stages may be 
cheaper in the long run than effectively repeating the quality exercise 
retrospectively. 


Presently the ASNA does not have a formal Social Accounting Matrix (SAM) 
associated with its core national accounts. However, the Income, Expenditure and 
Housing Section of the ABS have estimated household balance sheets by age class 
and socioeconomic group. This work is expected to be ongoing, so that time 
series data can be developed for comparative purposes, and so it is a candidate for 
OMSEA pilot testing for reasons similar to those mentioned in relation to the 
Tourism Satellite Accounts 1997-1998. 


Finally, there is being developed by ABS a major pseudo social accounting matrix 
known as the Measuring Australia’s Progress (MAP) project. This is a very 
interesting collection of data which presents principal indicators from the 
economic, social and environmental sectors for consideration in topical debate. 
MAP is not, however, strictly a social accounting matrix, because it doesn’t attempt 
to link its principal aggregates in some formal causal manner. However, the fact 
that MAP is not a SAM doesn’t mean it shouldn’t have a quality assessment. What is 
important here is that QMSEA has not been developed to assess the quality of 
systems dealing primarily with social or environmental activities. The MAP project 
is therefore not considered to presently be an appropriate candidate for case study. 
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3.3 Assembling the quality indicators 


Once the case studies have been selected for QMSEA testing, the various quality 
indicators appropriate to each system will need to be assembled. Depending on the 
particular case study this exercise will progress with differing degrees of ease. For 
those case studies that are primarily dependent upon ABS collected input series and 
transformation methodologies, the task of collecting and developing quantitative, 
qualitative and descriptive quality indicators is expected to be easier than for case 
studies dependent upon many input data streams not under ABS direct responsibility. 


However, to support a free ranging interrogation procedure of the type described in 
Box 4, large banks of quality information need to be collected and arrayed. Such an 
assembly of information is here called the quality matrix. A concern is whether it is 
feasible and necessary to fully populate the quality matrix. There is concern about the 
time, effort and opportunity cost, and the amount of resources required to be borne 
in an attempt to populate the quality matrix. This question is not easy to answer 
without the hindsight from trials and is partly determined by what is considered to be 
the primary aim of QMSEA. 


If the quality matrix is not sufficiently populated there is concern as to whether an 


informed judgement can be made about the “fitness for use” in various applications. 
For example, ifa principal SEA aggregate is a complex function of many variables its 
consequent quality may not be gauged or measured without knowing: 


° the quality attributes of most of the substantive component variables; 


° the salient features of the complex function that transforms the 
component variables; and 


° the conditional assumptions and judgements which constrain the two 
above elements. 


Axiomatically, ignoring trivial variables, trivial features of the complex function, 
and trivial assumptions should not impact adversely upon the exercise of 
producing quality indicators for the principal aggregate. But in practice 
determining what is trivial is not always clear cut in differing circumstances and 
given the variety of uses of the statistical aggregates. 


To some extent, how many cells should be filled in the quality matrix will be 
determined by what purpose primarily drives the adoption of QMSEA. For instance, 
satisfying a high-end Treasury user’s attempt to determine the quality difference 
between, say, quarterly percentage movements of seasonally adjusted chained volume 
GDP or NDP will probably require the data matrix be populated differently to 
satisfying an internal ABS management enquiry as to where to more cost effectively 
spend the budget on quality improvement of the NSO’s various SEAs. 
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It should also be noted that there are differing degrees of difficulty or ease in 
populating the quality matrix according to which facet and or dimension is being 
considered, and for which SEA or data item. Some metadata concerning facets or 
dimensions of quality will be common to all SEAs with only some exceptions. Here 
this quality metadata is referred to as global. By its nature much of this global quality 
information will be descriptive. The definitions, descriptions and relationships of 
facets of quality are examples of global metadata. Similarly, some of the quality 
metadata will be common to data items within a particular SEA, but may differ 
between various SEAs. Here this type of quality metadata will be referred to as SEA 
specific. For example, metadata regarding quality aspects such as accessibility and 
timeliness will generally be a common denominator within a SEA, and will also be 
descriptive. On the other hand some metadata concerning quality aspects will be data 
item specific. For example, aspects associated with accuracy, revisions, longevity, 
continuity, compliance and relevance will be conditional on the data item, and 
whether it is: 


° original, seasonally adjusted or trend; 

° stock, flow or “flock”; 

° level, movement, percentage movement, ratio, index number; and 
° dollar value, quantity, chain price or volume, etcetera. 


This metadata will tend to be statistical as well as descriptive. 


There will be a degree of evolution and modification of various aspects of the quality 
indicators in the pilot phase. This is expected to occur as comprehension increases as 
to how aspects of the quality attributes of the input data change as it is compiled and 
transformed into the final aggregates. It is expected, however, that for some SEA 
assessments various quality indicators will have less than complete information and 
will consequently indicate that “information is unknown” or “unavailable”, or is subject 
to anumber of caveats. This feature is expected to be more common to the facet of 
overall "accuracy" of many data items, both input and output. However, a knowledge 
of this factual, albeit lame situation may still be helpful in judging how SEA statistics or 
pieces of “intelligence” should be regarded for use in a particular application. 


3.4 Information issues and probity 


Electronic storage of the QMSEA quality measures 


It is the medium term goal of QMSEA to assemble, store, manipulate and access the 
quality metadata via an online e-databank. This approach to the knowledge 
management is recommended because the quantity of the metadata concerning 
aspects of quality, the need to update it in an irregular ad hoc fashion and the need to 
be able to retrieve and interrogate it in many different ways precludes hard copy 
storage and distribution as an appropriate medium. 
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Presently some quality indicators for some ABS data collections are stored in various 
ABS e-facilities, such as the Collection Management System and ABS Database. 
However, this coverage is by no means comprehensive with respect to SEAs. After 
user acceptance testing of the proposed flexible interrogation procedures (Refer 
above to Section 1.8 — Outputs from the QMSEA project) the existing e-facilities will 
require extensive development if QMSEA is to be promulgated for the gamut of SEAs 
and made available to the key external users. The development by ABS of a new input 
database warehouse may have an important and timely role to play in this area of 
QMSEA implementation. 


A summary of QMSEA measures for the lay user? 


There is a view that the quantity of information about aspects of quality in each of the 
large systems of economic accounts will be overwhelming for the general users of 
statistics, particularly those not making regular or intensive use of the accounts. Even 
for the sophisticated users there will be the issue of whether the amount of 
information concerning quality aspects is overwhelming. The question therefore 
arises as to whether the information about quality aspects should be summarised. At 
Appendixes B and C there are examples of draft Summaries of the IMF’s Data Quality 
Assessment Framework. Appendix B is referred to as the “lite” version of a DQAF. To 
the above there is a counter view that such summaries are too superficial to be 
helpful, and that important weaknesses in SEAs may be hidden under the gloss or 
veneer of summary information. This aspect is discussed briefly later. 


Risks in providing Summary measures 


There is also the issue and risk that highly summarised or global composite indicators 
of SEA quality will encourage finance analysts to assign performance rankings to these 
types of quality measures, with unwarranted and unfavourable consequences. The 
IMF has reported that some users in the financial markets have already requested of it 
a form of DQAF that would permit a country ranking or scoring system for data quality 
aspects. The IMF has responded by claiming that: 


“However, the data quality assessment framework does not lend itself to such an 
approach. The element of subjectivity inherent in the frameworks, the detail embedded 
in the dataset specific frameworks, and the great diversity of country circumstances 
largely preclude using them to make meaningful country rankings.” (IMF/Carsons, 2000, 
para. 37 p. 15) 


I doubt whether such an explanation will satisfy or inhibit the financial analysts from 
their desire to have a simple rating, and ’'m not an advocate of the summary, “lite” or 
composite indicator approach to QMSEA quality assessment. 
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BOX 10 — HOW MUCH DETAIL IN THE QUALITY MEASURES? AN ILLUSTRATION 


The difficulty of deciding how much detail is appropriate for quality assessment can 
be illustrated by considering the principal economic indicator:— quarterly 
percentage growth of the seasonally adjusted chain volume measured Gross 
Domestic Product. This version of GDP is clearly one of the most complex, 
composite and synthetic statistical measures of economic activity. How then 
should the quality of this key indicator from the system of national accounts be 
described? Given this statistic is a high order aggregate that is the result of many 
data input streams, that are in turn transformed by both an elaborate seasonal 
adjustment methodology and a complex chain linking volume methodology, does 
its nature defy being described as simply either “good”, “fair” or “poor”? 


Further, is Gross Domestic Product (GDP) or Net Domestic Product (NDP) the 
relevant measure for analysts to focus on now and for the new economy? How 
should the QMSEA measure of quality, in particular “relevance”, attend to this issue 
summarily? Consider the issue briefly. The term “Gross” in Gross Domestic 
Product indicates that the estimate includes the value of fixed capital consumed. 
On the other hand, NDP is derived by subtracting from GDP the consumption of 
both tangible and intangible fixed capital in the production of Domestic Product for 
the period considered. Compared to GDP the NDP measure may be the more 
conceptually appealing and satisfying measure of economic activity for analytical 
purposes. 


Although it may be argued that NDP is a better quality indicator than GDP, because 
it is more relevant or elegant for economic analysis, not all the other quality aspects 
are equal between these two important measures of macro economic performance. 
Because the estimation of the consumption of fixed capital is problematic GDP is 
thought to be generally the more accurate measure compared with NDP; GDP can 
be estimated in principle from three independent sources. However, it should be 
acknowledged that presently a significant gap in the source data for quarterly 
national accounts is capital expenditure on computer software and that this 
deficiency has implications for the accuracy of both measures of Domestic Product. 
In the new economy this is a large and growing component of Gross Fixed Capital 
Expenditure, and therefore a problematic area for GDP as well as NDP with regard 
to the value of fixed capital consumed. 


There are a number of important quality issues to consider in this example. 
Evidently, making an informed decision between these two indicators with regard 
to “fitness for use” requires comprehending more than just a composite summary 
quality indicator. 
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One of the challenges in assembling and presenting the information about quality is 
ensuring that the aspects in which a user is interested or should be aware of, are 
conveyed so that the user can decide whether the statistical output is fit for their use. 
The difficulty of deciding how much detail is appropriate for quality assessment is 
illustrated in Box 10. 


Self-assessment of quality? 


Another concern is that if the quality information is not presented at a sufficiently fine 
level of detail the resulting generalisations and summaries may hide, either 
intentionally or inadvertently, important deficiencies in the statistics or the SEA’s 
methodologies. Concern has been expressed already that some self assessments 
using the IMF’s present frameworks are unlikely to expose weaknesses in some 
country's system of economic accounts. This aspect involving sufficient detail about 
quality features is a major concern for developing credible DQAF/QMSEA type systems 
for practical use. 


In the Foreword to the ABS Corporate Plan 2000 it is stated that: 


“This Corporate Plan reaffirms the ABS Mission that we serve the whole community and 


that a good statistical system is important pillar of a robust democracy.” 


Judgements about how good a statistical system is requires evidence concerning the 
quality of its major systems of economic accounts, as well as its other statistical 
products and services. Evidently, to be useful the quality information about SEAs 
needs to be credible. The question therefore arises as to whether it is appropriate to 
require any NSO to conduct its own quality assessment when it has a vested interest in 
the outcome? The ABS Corporate Plan 2000 states: 


“We are also open about the quality of statistics, so that users can better understand and 


interpret them.” 


But how open is open? Would there be concern, for instance, from quarters either 
within or outside the ABS that publication of information about low response rates for 
some ABS surveys might encourage future respondents not to respond, and/or put 
the ABS under pressure to explain why its Notice Of Direction policy is not effective? 


There is therefore the issue of whether a QMSEA quality audit of SEAs be regarded as 
a type of open “performance” audit, involving a consortium of key user agencies such 
as the Treasury, Reserve Bank of Australia, and Prime Minister and Cabinet. Such a 
consortium could provide a useful role for key users to assist ABS in 
applying/assessing the QMSEA quality information, thereby providing directly a user 
perspective on those aspects of quality that are important to them in the use of SEAs. 
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4. CONCLUSION 


Regarding other data quality assessment procedures being developed in ABS and 
overseas QMSEA differs in some important respects. Some examples of differences 
are — 


1. QMSEA is designed to investigate whether present and future systems of 
economic accounts (SEAs) are likely to be “fit for purpose”. 


2. | QMSEA looks beyond probability based sample survey data to the full range of 
input data used to compile SEAs. 


3. QMSEA will concord with the IMF Data Quality Assessment Framework, an 
international standard being developed, but aims to be more comprehensive 
than DQAF — in both the breadth and depth of quality auditing. 


4. QMSEA is designed primarily for a relatively small, sophisticated group of users 
(both outside and inside the ABS). 


5. Assessments against the QMSEA standards may in the long run use a 
combination of ABS self-assessment and external assessment (by, say, a 
consortium of user agencies such as the Treasury, Reserve Bank of Australia, and 
Prime Minister and Cabinet). 


6. | QMSEA’s quality measures may be made available in electronic form, so they can 
be readily maintained and interrogated. 
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Government Finance Statistics 
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GLOSSARY — TWELVE FACETS OF QUALITY 


The following terms used to describe facets of quality in Quality Measures for Systems 


of Economic Accounts (QMSEA) are not necessarily independent or mutually 


exclusive of each other in their meaning. 


Each explanation of a term in this Glossary provides generally a brief statement of the 


term’s intent within the context of QMSEA. A definition or description of what the 


term means covers conceptually then follows. Where appropriate an indication of what 


is not meant by the term is given, and these instances should be noted. Examples 


from various Systems of Economic Accounts (SEA) are cited to draw distinctions or to 


illustrate interdependency with other QMSEA terms where appropriate. 


Accessibility 


36 


Accessibility here relates to data and information dissemination matters. It 
considers the degree to which the SEA input/output data and related 
information are available to users. Aspects covered by the term accessibility 
involve data availability, metadata availability and the degree of interpretative 
assistance available to users of the particular SEA provided by the NSO. 


Data availability involves issues of data presentation and cataloguing, distribution 
medium, as well as the availability of non-confidential but unpublished data used 
in the SEA. Metadata availability is related to issues surrounding the availability 
of information concerning the concepts, sources and methods associated with 
the SEA, including the availability of a comprehensive set of quality indicators for 
the SEA. 


Accessibility also considers whether the information surrounding or supporting 
the SEA is openly available to the public, or whether some aspects of the SEA 
data and/or methods are not made public for reasons other than respondent 
confidentiality, for instance, because they are perhaps considered embarrassing 
or sensitive areas. For example, is some information concerning non-response 
rates in sample surveys suppressed in order to prop up confidence in the survey 
results and/or avoid awkward questions regarding the application of serving 
Notices of Directions to recalcitrant respondents? Or are details of specific 
influential items in the Consumer Price Index basket of goods suppressed in 
order to avoid manipulation of outcomes by some respondent retailers? 


Accessibility is also related to lucidity. Lucidity considers how understandable 
the metadata information is likely to be to users once access to it has been 
gained, whereas accessibility considers the ease with which that information may 
be obtained from the distributional mediums or vehicles, regardless of whether 
it is comprehensible. 
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Accuracy 


Accuracy here is concerned with the proximity of an estimate to the ideal or 
true, but unknown value of that component. It is concerned with the degree of 
precision associated with the estimate. 


It is important to note that the true value is never known and that in practice 
there cannot be an overall measure of accuracy or total error. Consequently, 
accuracy is evaluated by considering the potential sources of error and 
considering whether they have been minimised optimally. 


Accuracy is distinguishable from the terms reliability and revisability in the 
following way. The term reliability in the QMSEA context refers to the degree 
that an estimate or SEA information rates against all of the facets of quality, not 
just the single facet of, say, accuracy. That is, an estimate is said to be more 
reliable than another if it rates more highly against all of the QMSEA criteria than 
the alternative estimate. In comparison, revisability refers to the characteristics 
that the initial and subsequent intermediate estimates exhibit as they evolve 
towards the final or stable estimate of a component for a particular time period. 
It is possible for an estimate to be inaccurate but subject to no revision. 


The degree of accuracy of an estimate, at a point in time and over time, is 
determined by a number of influences. Errors affecting the degree of accuracy 
can be classified as follows: survey sampling errors; frame errors; measurement 
errors; processing errors; non-response errors; model assumption errors, 
etcetera. The following list is not ranked in order of importance, nor is it 
necessarily complete. 


Input data collection errors 


These may involve the information providers or respondents being unable to 
report on the required basis, unintentional or intentional mistakes in reporting 
or late reporting, and errors occurring in the recording and processing of the 
input data reported. In SEAs there may be up to six input data streams so 
impacted: complete enumerations, administration or transaction by product 
collections, probability sampling, non-probability sampling, market/media 
intelligence and a synthesis of some or all of the previous streams. These 
streams are sourced from either the Australian Bureau of Statistics (ABS), the 
rest of Australia (ROA), or the rest of the World (ROW). Some of these streams 
and sources are more credible or accurate sources than others, in that the have 
established proven track records over time, and some are more problematic in 
that they are traditionally more difficult to measure. 
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Inaccuracies in the time of recording and the valuation of stock and flow items 


Considered here is whether the time of recording “transactions” is in accordance 
with the principles of accrual or full accrual accounting, etcetera. Whether the 
valuation rules are appropriate and consistent in application across the SEA. For 
instance, market price rather than transfer price is used as the basis of valuation 
of market transactions in systems of economic accounts. A consideration is the 
extent to which output prices are recorded as either basic prices or producers’ 
prices. Basic prices are exclusive of transport and trade margins, of taxes on 
products like value added taxes, and of subsidies on products. Producers’ prices 
differ from basic prices in that they do include subsidies and taxes on products. 


To be consistent with the valuation of domestic production, imported and 
exported goods should be valued at the border on a “free on board” (fob) basis. 
However, imports may be valued at the border of the importing country, rather 
than the exporting country, at a cost, insurance, freight basis for the purpose of 
compiling the supply and use tables at the disaggregated level of individual 
products. 


Whether income and service transactions are recorded gross of any withholding 
taxes deducted at source, and securities transactions are recorded gross of any 
applicable fees and commissions. 


Whether foreign currency transactions are converted into the domestic currency 
accurately. Transactions in foreign currency should be converted using the 
midpoint exchange rate prevailing in the market at the transaction date. If not 
the next best solution is to use the average exchange rate for the shortest 
possible period when that actual exchange rate is not available. 


Where transaction estimates are derived from stock data, to what extent is an 
attempt made to revalue the stock data into their original currencies and then 
convert the change in original currency to domestic currency at the average 
exchange rate of the applicable period? 


In non-market transaction the cost approach may have been adopted. Where 
barter transactions are a consideration how accurate/applicable have observable 
“prices” been in similar/comparable situations. If no prices are observable 
valuation at current cost may be deemed the next best solution, followed by 
historic costs and, if nothing else is available, the discounted present value of 
expected future returns. 
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Methodological errors associated with the input data collection vehicle or 
compilation procedure 


For example, if the input data are derived from scientific probability sampling 
techniques, such as stratified sample surveys, the sample point estimates will 
have associated with them a Sampling Distribution Standard Deviation (SDSD), 
frequently referred to as the “sampling error”. Note that this “sampling error” 
doesn’t attempt to measure all errors associated with such sample results, even 
though the term “sampling error” might imply that it does cover all sampling 
errors. In the QMSEA framework the term “sampling error” will be infrequently 
used, so as to avoid confusion with the phrase “errors associated with sampling”. 


The SDSD can be used to construct various confidence regions about some 
sample point estimates, under various conditions or assumptions. The smaller a 
SDSD is for a sample point result the more accurate it is thought to be, all other 
factors held constant. It should be noted that not all sample results have 
associated with them SDSD values. An example of surveys which don't produce 
SDSD values are the non-probability or purposive surveys used for producing 
the Consumer Price Indexes and Producer Price Indexes (Price indexes for 
materials used in, and articles produced by manufacturing industries, export and 
import price indexes). 


For the ASNA no SDSD have been developed generally for the aggregate output 
data or key balancing items, like Gross Domestic Product or Private Final 
Consumption Expenditure. SDSDs for these “headline” indicators are unlikely to 
be developed because of the nature of the compiling procedures used in the 
SEA to transform the input data streams into the SEA output data, in a supply 
and use balanced integrated manner. 


Other methodological errors arise in the areas of chain volume estimation, price 
index construction, seasonal adjustment and trend estimation. In these and 
other similar methodological areas the degree of accuracy may be investigated 
by considering axiomatic and non-axiomatic criteria, both in the theoretical and 
practical domain via simulations. Such results are often only indicative and 
should not be taken as definitive quality indicators. 


Compliance 


Compliance here refers to the degree or extent to which in practice the 
principles or objectives of the particular SEA, or methodology in question, are 
upheld or met. It should be noted that with regard to compliance a National 
Statistical Office could rate well against a particular SEA, say the System of 
National Accounts 1968, however, in the year 2001 this NSO’s SNA 1968 based 
SEA would rate poorly against the quality facet of relevance. 
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Similarly, a particular type of methodology may be employed by a NSO 
consistently and with lapidary precision and conscientiousness, indicating a high 
compliance rating. However, the methodology may be known to produce less 
than accurate results, or be inferior conceptually to another methodology, or 
also be subject to a high degree of revision. The ABS use of “constant price” 
estimation prior to the adoption of the superior chain linked volume method, is 
an example. On the other hand, a NSO may claim to have adopted a particular 
SEA in principle, say SNA 1993 or ESA 1995, but in practice have not substantially 
implemented the preferred system for various technical and non-technical 
reasons, and therefore rate less favourably against both quality aspects of 
compliance, consistency and relevance. 


The aspect of compliance may impact on quality aspects of accuracy, relevance 
and consistency. 


Consistency 


40 


Consistency here is concerned with the conceptual integrity of the SEA or its 
transforming methodologies. That is, for a SEA to what extent is its framework, 
definitions, classifications and conceptual boundaries logically sound, 
unambiguous and complete internally. It should be noted that theoretically a 
particular SEA may achieve a high degree of consistency but be judged to be not 
so relevant to contemporary requirements of analysis or decision making. For 
instance, note the supplementation of the core accounts of the ASNA with ad 
hoc satellite accounts and social accounting matrices. Consistency also 
considers the degree to which a SEA is conceptually consistent or integrable 
with other comparable SEAs, either domestically or internationally, e.g. SNA 
1993, BPM5, GFS 1986, etcetera. 


Consistency is also a quality aspect of a SEA’s transforming methodologies. For 
instance, the commonly employed X—11 based seasonal adjustment 
methodology is not consistent between monthly and quarterly flow data. Nor is 
it consistent in some other esoteric areas such as time reversal, or time labelling 
of the analysis span (ignoring trading day factors). Chain indexed volume 
measures are not universally consistent for “aggregation” purposes. For 
example, unlike constant price estimates, they are not strictly additive, in that 
the sum of the chain volume estimates for the components of some principal 
aggregate, such as GDP or total consumption, may differ from the chain volume 
measure of the total. Also, various types of price index methodologies are not 
consistent with respect to the full gamut of axiomatic criteria. Some of the 
inconsistencies are trivial, and some may not be for various forms of analysis. 


ABS * QUALITY MEASURES FOR SYSTEMS OF ECONOMIC ACCOUNTS * 1352.0.55.041 


ABS METHODOLOGY ADVISORY COMMITTEE * JUNE 2001 


The quality facets of consistency and continuity are similar aspects in the QMSEA 
context. Consistency has a point-in-time dimension in the QMSEA context, 
whereas continuity has an across time dimension. 


Continuity 


Continuity here has a temporal dimension, and is concerned with the extent that 
the conceptual consistency of the SEA framework, or the time series structural 
nature of the input data, or methodological impact on data output is sufficiently 
the same or continuous over time, so as to permit meaningful period to period 
comparisons, analysis and econometric modelling of socioeconomic 


relationships. 


Continuity may be influenced by changes in the collection procedures of input 
data, changes to the concepts and definitions of the particular SEA, changes to 
the compiling/transforming methodologies, and/or changes to the time series 
characteristics of the data brought about be socioeconomic factors. Examples of 
the type of time series changes brought about by the above are, rapidly changing 
seasonal or trading day patterns, dislocations in trend levels or abrupt changes 
and reversals in rates of trend growth, or varying degrees of irregularity. 


The quality facets of consistency and continuity are similar aspects in the QMSEA 
context. Continuity has an across time dimension, whereas consistency has a 
point-in-time dimension in the QMSEA context. 


Frequency 


Frequency here relates to the periodicity of the SEA’s data reference period. It 
considers whether the reference periodicity is annual, quarterly or monthly, 
etcetera, and whether the particular periodicity is adequate to permit 
meaningful cyclical analysis by the key users. In this regard it is important to 
note that an economic cycle of duration shorter than twice the time interval 
between two contiguous data observations, or reference periods, cannot be 
properly recognised. Such a cycle will be statistically confused with a cycle of 
longer duration. Consequently, if the data reference period is annual, only 
cycles of duration longer than two years in length can be observed from this set 
of data, even though the data may be generated by cycles shorter than two years 
in length. Expressed another way, if cycles of length one quarter are to be 
monitored in practice the reference period has to be no longer than six weeks. 


It should be noted that the periodicity of the data reference period is not 
necessarily the same as the periodicity of the so called “collection process”. For 
instance, data whose reference period is a quarter may be collected yearly, or 
monthly data may be collected quarterly. 
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The quality aspects of frequency, relevance and timeliness and accuracy can have 
bearing on each other. 


Longevity 
Longevity here refers to whether the SEA data contains sufficient time series 
observations to undertake meaningful analysis of a historic nature, or whether it 
is sufficiently long to permit a particular statistical methodology to be 
appropriately applied to the data, such as seasonal adjustment or econometric 
modelling. For seasonal adjustment five years’ worth of observations are usually 
needed to produce estimates of basic useability. More observations may be 
required if trading day variation is an influence in the data and needs to be 
estimated. 


It’s possible for a SEA to have short time series that are continuous, whereas 
another SEA may have long time series that may not be very continuous. 


The quality facets of longevity, continuity, relevance, accuracy and revisions are 
often related. For instance, to maintain relevance, continuity may be given a 
lower priority, leading to shorter time series each having by definition reduced 
longevity, and in practice lower accuracy and increased revisability of the 
seasonally adjusted data. 


Lucidity 
Lucidity here is the facet of quality which relates how well users are serviced by 
the information about the SEA, and assisted with data interpretation and 
analysis, so that the SEA output may be understood and used effectively and 
appropriately. Lucidity here considers the informational content of “concepts, 
sources and methods” documentation, and the usefulness of “main features” 
commentaries and “explanatory notes” provided by the NSO. Lucidity is 
concerned with the “educational” value of this information. 


For instance, in seeking to assess the difference in quality aspects of GDP viz 
NDP there is a qualitative difference between being told: 


“GDP less the consumption of fixed capital is called net domestic product.” (ABS, 
2000, ABS cat. no. 5216.0, p. 30) 
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compared to being told: 


“To ensure that GDP is a measure of economic production that is free of 
duplication, national accountants draw a distinction between intermediate and final 
purchases. All goods and services which are used up in the course of production 
by resident producers are intermediate purchases (and are referred to as 
intermediate consumption). Purchases of capital goods and finished goods and 
work in progress going into inventories are the only purchases of goods by 
resident producers that are regarded as parts of final demand. The ‘gross’ in GDP 
indicates that no deduction is made for consumption of fixed capital; in other 
words, the gradual using-up of the economy’s productive assets is not accounted 
for in GDP. A product figure net of capital consumed (i.e. net domestic product) 
would recognise capital consumption as a necessary cost of producing the 
economy's goods and services and would therefore be a purer measure of 
production. However, because of the difficulties involved in measuring capital 
consumption (and more particularly variations in methods used between 
countries), GDP has remained the major summary measure of economic activity.” 
(ABS, 2000, ABS cat. no. 5204.0, p. 90) 


This facet of quality is related to, but different from the quality facet of 
accessibility. Lucidity considers how understandable the metadata information is 
likely to be to users once access to it has been gained, whereas accessibility 
considers the ease with which that information may be obtained from the 
distributional mediums or vehicles, regardless of its degree comprehensibility. 


Relevance 


Relevance here refers to the ability of the SEA information, and “intelligence”, to 
satisfy contemporary and likely future needs of Australia, both nationally and 
internationally. These needs are concerned with measuring and describing the 
socioeconomic concepts and aspects with which key users are most interested. 
Relevance is here concerned with how well the SEA is providing information 
concerning the most important features and developments about the various 
socioeconomic phenomena it is intended to address for decision making. 


The facet of relevance is related to timeliness in that if the SEA information or 
intelligence is not available soon enough after the reference period for a 
particular use it is likely to be regarded as less relevant for analytical purposes. 
Refer to timeliness for a further discussion of this aspect of quality. Similarly, the 
facet of relevance is also related to the facet of frequency. If the frequency of a 
SEAs reference period is too low, important economic cycles may be 
misinterpreted or reacted to too slowly, and the data may not be considered 
very relevant for some types of analysis. 
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The term relevance also considers temporal aspects, such as whether the annual 
data from the SEA is calendar or fiscal year based. If the data are fiscal year 
based, the issue is whether the fiscal year is of a type that is relevant for 
international comparisons, as not all countries have matching fiscal years. 


Similarly, stock measures such as assets, liabilities, employment or 
unemployment, and prices, have a temporal features that are important for 
relevance considerations. These stock indicators should in principle be 
measured at a point in time rather than over an interval of time. In practice 
some of these stock indicators are not collected at a point in time. Some are 
collected over a time interval, with the measurement or reference interval 
drifting over time, rather than being fixed at the start of the reference period. 
For instance, employment and unemployment numbers in Australia do not 
relate to a specific time point each month, but are collected over the period of a 
fortnight and relate to the fortnight before collection. The interval between 
collection periods is approximately monthly, but has been as short as three 
weeks and as long as six weeks. This drifting reference interval of the labour 
force stock indicators may add a degree of artificial variability to the raw data. 
Also, some stock indicators are constructed from data collected at time points 
spread throughout reference period. For example, the Consumer Price Index is 
not a composite index of prices all measured at the same point in time: its 
constituent price components are collected throughout the quarter. These 
features of the data can have important ramifications for whether these SEA data 
are relevant for various forms of analysis. 


The above aspects of relevance are distinguished from accuracy in the QMSEA 
context in the following way. The above stock data may have been collected and 
compiled in strict accordance with a set of design specifications, and so may be 
referred to as accurate with respect to the design specifications. However, it 
may be that the design specifications have led to the production of data that is 
less relevant for some analytical purposes. 


The term relevance also considers whether there are questions of statutory 
relevance. For instance, the Australian Statistician has suggested in June 2000 
that various price index methodology should be enshrined into legislation only 
once the pros and cons of the indexes are understood by the users. 


“It was suggested that users be educated about the advantages and disadvantages 
of using different indexes for contract escalation purposes. When a price index is 
to be used by the government as a price escalator, Mr McLennan said that it is 
important that the ABS strongly encourage Ministers to ensure the correct index is 
enshrined in legislation.” (Prices, Productivity and Public Finance, DHM/2000/, 
para. 13) 
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Revisability 
Revisability here does not mean the procedural degree of ease with which the 
SEA, its attendant methodologies/procedures, or its input/ouput data may be 
revised. The term revisability here refers to the revision characteristics which an 
estimate of a socioeconomic variable displays over time as subsequent estimates 
for the same reference period is made. It also considers the inherent revisability 
characteristics of the compilation/transformation methodologies employed with 
a SEA. For instance, the unavoidable revisions encountered with some chained 
link volume measures, seasonally adjusted series, and trend estimates. 


In the QMSEA context revisability is used in a similar sense to OMB’s Directive 
No. 3, in that for a particular estimate it considers the magnitude and direction 
of revisions, and these attributes of input and output data are relatively easy to 
measure compared to aspects of accuracy. 


The magnitude and characteristics of revisions to SEA output data have been 
suggested by some analysts as one of the most useful indicators of quality. While 
intuitively it make sense to regard data that is subject to large and generally 
one-sided revisions as being less useful than data subject to fewer revisions, the 
absence of any obviously large errors in a balancing item, such as GDP, may 
mean that there have been large compensating errors made on both sides of the 


accounts. 


Also, recent studies are finding that some users of the data appear indifferent to 
data revisions. For instance, Brooks ef al. (1999) found that while nine 
Australian financial markets do respond to new data, they do not respond to 
data revisions in the first instance. 


“A plausible reason for this result is that official announcements simply confirm or 
deny market expectations that are already discounted into prices. Given that 
markets are forward looking, revisions of ancient history are less relevant than 
current data. The news component of the announcement ‘crowds out’ the 
revision. Alternatively, the often massive revisions that are made may partially 
discredit the extent of the revision and markets may simply disregard them.” 
(Brooks and Davidson, 2000) 


Revisability in the QMSEA context also considers the effect of various arbitrary 
revision policies that may be adopted for operational convenience by the NSO. 


In the QMSEA context the terms accuracy, revisability and reliability relate to 
different aspects of quality. 
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Sense 


Sense here considers the degree to which the SEA input/output data “makes 
sense” or accords or behaves coherently with other partial, leading, concurrent 
or lagging indicators or signals from other datasets. It is a measure of quality 
that also encompasses data input and output credibility checks. Such checks 
may, for instance, involve judging whether the seasonal patterns displayed by 
new accrual based Government final consumption data accord with expectations 
that the patterns be weaker than that displayed by the cash based data. 


The supply and use balancing approach in the annual ASNA is aimed at ensuring 
that the system of national economic accounts makes better sense. The supply 
and use approach to the compilation of the ASNA forces “reconciliation” of the 
production, expenditure, and income accounts. These types of accounts are 
traditionally compiled using independent sources and methods and can produce 
markedly different results. Generally the more disaggregated the level at which 
the “supply and use” method is applied, the more accurate the results tend to 
be. Such integration of the various datasets of the accounts should tend over 
time to improve the reliability (in the broad QMSEA sense) of all the datasets 
involved by exposing and eliminating inconsistencies which are indicative of 
errors somewhere in one or more of them. With regard to the quarterly ASNA 
the situation appears different to some degree. 


“In the national accounts user review the ABS received praise for improving the 
coherence between quarterly national accounts and associated partial indicators 
that are published ahead of it. The ABS was also urged to reduce, and preferably 
eliminate the statistical discrepancies for quarters and the last year. Ironically, 
quarterly supply and use tables would achieve a reduction, or elimination of 
statistical discrepancies at the expense of coherence with partial indicators.” 
(Aspden, C., BR/2001/7(Apr)) 


From the above it should be noted that while the extent to which a set of 
statistics appear sensible or are coherent, may be reassuring in that it is 
suggestive of greater accuracy, this attribute is only a guide. 


“.., it should be noted that a coherent set of statistics is not necessarily an 
accurate set, as the statistics that are being compared may suffer from similar 
magnitudes of error with the errors being in the same direction." (ABS, 2000, ABS 
cat. no. 5216.0, p. 400) 


Sense, consistency, continuity, and accuracy are related aspects of quality. 
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Timeliness 


Timeliness here is intended to measure the extent to which the time value of the 
SEA information is preserved. It recognises that the value of the SEA data for 
decision making tends to decrease as the time from the reference period 
increases. Also, given that headline SEA data tends to have significant 
commercial value and can affect foreign exchange, financial and commodity 
markets markedly, prompt release of SEA data may reduce the chance of 
unauthorised premature disclosure of the data and the adverse consequences 
associated with such events. 


Timeliness here is similar to the term used in the USA Office of Management and 
Budget’s Statistical Directive Number 3, Compilation and Release of Principal 
Federal Economic Indicators. (OMB, 1985) 


This facet may relate to other facets such accuracy, revisability, sense and 
relevance, access, and lucidity. An improvement in one these facets may 
improve or attenuate one or more of the other facets. 


Some specific aspects, criteria or standards of timeliness are as follows. 


a. The time interval between the reference period of the data/estimates and 
the public release date of the information should be as short as practicable, 
all other things being equal. As a standard, data issued quarterly or 
monthly should be released within XX working days of the reference 
period. See b. 


b. | Adata release schedule should be published for XX months ahead, and 
indicate a regular cycle for release of particular sets of SEA data. In 
addition, individual publications (hardcopy or electronic) should include 
notification about the next release date and time. Changes to this schedule 
should only occur if unforeseen and extraordinary circumstances arise and 
such changes must be broadcast in advance of the scheduled release date 
and fully explained. 


c. To what degree are release dates of relevant SEAs in synchrony, and vice 
versa, with the analysis schedules of key users, such as Treasury and 
Reserve Bank's fiscal and monetary policy formulation cycles, and for 
keeping the general public informed, and attuned to the dissemination 
realities of the media. 


d. Where interpretation of the SEA data may be affected by planned changes 
either to source data, the conceptual framework of the SEA, or the 
methods of estimation or compilation, such changes should be announced 
as far in advance as possible, but at least XX months in advance. 
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APPENDIXES 


A. QMSEA ENVIRONMENT — SCHEMATIC OUTLINE OF THE FEATURES 
OF SYSTEMS OF ECONOMIC ACCOUNTS 


Appendix A is available from the author on request. 
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B. INTERNATIONAL MONETARY FUND — DATA QUALITY 
ASSESSMENT FRAMEWORK - “LITE” SUMMARY 


Appendix B is available from the author on request. 


Revised (1993) version: http://dsbb.imf.org/vgn/images/pdfs/dqrs_factsheet.pdf 
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C. INTERNATIONAL MONETARY FUND — SYSTEM OF NATIONAL 
ACCOUNTS — QUALITY ASSESSMENT FRAMEWORK. 


Appendix C is available from the author on request. 


Revised (1993) version: http://dsbb.imf.org/vgn/images/pdfs/dqrs_nag.pdf 
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D. QMSEA AND INTERNATIONAL STANDARDS 


A feature of the IMF quality project DQAF is that there is a relatively close mapping to 
some existing international statistical standards, concepts and manuals. For example, 
the 1993 System of National Accounts and the fifth edition of the IMF’s Balance of 
Payments Manual are used as benchmarks for definitions, concepts, classifications and 
valuation methodologies. ABS generally complies with these standards and so the IMF 
quality framework provides a basis or foundation on which to build more 
comprehensive quality assessment system for the ABS's economic accounts. 


However, there are some concerns associated with attempting to base a quality 
assessment system solely around existing international standards. International 
standards are not necessarily monolithic, homogeneous, consistent, generally 

complied with, integratable, or best current practice, as discussed briefly below. 


Internationally accepted statistical standards can be useful, but they are not always the 
currently best available practice. For instance, consider SNA 1993, which is the 
present standard adopted by the ABS for its national accounts. This version of SNA, 
however, followed SNA 1968, and was implemented by the ABS in December 1998 for 
the September quarter 1998 national accounts. It was changes in the structure and 
nature of the economy, the increasing sophistication and growth of financial markets 
and their instruments, emphasis on the interaction of the economy with the 
environment and other considerations, such as a need to clarify and simplify the 
prevailing accounting system, that indicated a need to update the SNA from the 1968 
version. 


The task of modernising SNA commenced in the mid 1980s and the new version was 
released under the auspices of the Commission of the European Communities (now 
Eurostat), the International Monetary Fund, the Organisation for Co-operation and 
Development, the United Nations and the World Bank in 1993. Until the 
implementation by ABS of SNA 1993 in 1998, the system used through the 1990s was 
based on the conventions and practices prior to 1968 and evidently not up to date. 


Recently there has been much talk about the so-called “new economy”. 


“The term New Economy refers to a set of qualitative and quantitative changes that, in the 
last 15 years, have transformed the structure, functioning and rules of the economy. The 
New Economy is a knowledge and idea based economy where the keys to job creation 
and higher standards of living are innovative ideas and technology embedded in services 
and manufactured products. It is an economy where risk are uncertainty, and constant 


change are the rule rather than the exception.” (Edwards, 2001, para. 22) 


This raises the question of whether SNA 1993 is relevant or sufficient to the task of 
monitoring the new economy now or about 2010. 
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In the mid 1990s the OECD was told: 


“More seriously, I believe that by most criteria US economic indicators meet or exceed 
reasonable criteria for timeliness and also meet those criteria for reliably measuring 
short-term economic change. This, however, is quite different from saying that our 
statistics are keeping pace with longer run forces shaping the US, indeed the world, 
economy. Our economies are changing faster than our ability to measure them. 
Economic output is increasingly conceptual rather than physical. The form of output and 
the means of production have been increasingly less physical and tangible, hence more 
difficult to measure. ... This trend is sure to continue into the 21st century.” (Waite, 
1995, p. 10) 


In Australia, as elsewhere, commodities and industries grow at different rates, and new 
products come onto the market and old ones leave on a regular basis. It is therefore 
important that our SEAs are flexible enough to keep pace with the change. 


Unlike DQAF, one aim of QMSEA will be to explicitly assess and review whether the 
SEA methodologies and basis are relevant and appropriate for the present and 
foreseeable future circumstances of Australia. QMSEA will not automatically assume 
an international standard is necessarily the best or appropriate benchmark for the 
above reasons and those reasons discussed below. 


Some international “standards” are not monolithic or homogeneous in their 
application, or universally adopted by NSOs. SNA 1993 is an example. It was designed 
to be flexible in its operation so that economies in various stages of development or in 
particular economic/financial circumstances could be accommodated under a broad 
accounting umbrella. However, SNA 1993 is still not a system adopted by all 
countries. This aspect of SNA 1993 is illustrated by the fact that all the countries of the 
European Union don’t use SNA 1993 but rather use the system known as the 
European System of Accounts 1995 (ESA 1995). This system is almost consistent with 
SNA 1993, and so is similar to, but different from the SNA 1993, the differences 
catering for specific European requirements. (McCarthy, 1999) It needs to be 
recognised that in some circumstances parts of an international standard, like SNA 
1993, may not be suitable for Australian systems of economic accounts or conditions. 


Some of the international “standards” are not consistent or integrated with each other. 
This is ironic, as one of the main reasons for setting international standards such as 
SNA 1993 or ESA 1995 is to maximise international comparability of these important 
statistics. Several international accounting frameworks are used by NSOs in addition 
to SNA 1993 or ESA 1995. Two examples are the Balance of Payments Manual, Sth 
edition, 1993 (BPM5) and the Government Finance Statistics manual 1986 (GFS 1986). 
The GFS 1986 is not consistent with the conceptual framework of either the SNA 1968 
or SNA 1993 or ESA 1995. The inconsistencies arise mainly because SNA/ESA data are 
compiled as closely as possible on an accruals basis while the GFS 1986 largely records 
transactions on a cash basis. Although there have been moves underway to update 
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GFS 1986 to ensure greater consistency with SNA 1993 and ESA 1995 in valuation and 
other conceptual areas, it will not be possible to eliminate all differences. 


Another complication for effective integration and consistency between the SEAs is 
that although, for example, the GFS may be changed to an accruals basis theoretically, 
the availability of source data primarily on a cash basis still gives rise to inconsistencies 
between the systems in practice. This can occur for contemporary periods as well as 
historic periods. For instance, although the Commonwealth Government of Australia 
shifted to an accrual accounting basis on 1 July 1999, there were subsequent delays in 
receipt of accrual data by the ABS that have lead to subsequent delays in the release of 
the quarterly national accounts for four quarters. 


“.., the move to accrual accounting has not been without its problems. There have been 
many issues associated with the provision of accrual data from the various jurisdictions 
that have required resolution and, in some cases, the data available for national accounts 
compilation has been far from ideal. For two State Treasuries — Queensland and Western 
Australia — data simply have not been provided and they have had to be estimated by the 
ABS using budgetary information. Also, as it has generally not been possible to 
re-estimate historical series on an accrual basis, there have been issues associated with 
seasonal adjustment due to apparent changes in seasonality that cannot properly be 
quantified.” (Harper, P., BR/2000/5 (Apr), para. 16) 


Here there is a practical example of the tradeoffs and interactions between the various 
facets of quality: accuracy, relevance, consistency, revisability, longevity and timeliness. 


The unavailability of source data can also lead to parts of a particular system of 
economic accounts being inconsistent, besides leading to various SEAs being partly 
inconsistent with each other. For practical reasons complete implementation of 
accrual accounting has not been possible in some other areas of the ASNA. Some 
examples of where accrual accounting has not been adopted include household 
income tax and various employee entitlements including recreation and long service 
leave. 


While many countries of the world have expressed their intentions to produce their 
Statistics in an SNA1993 framework and format, this by no means guarantees that their 
data conform to the SNA 1993 standards. 


“Without some assessment of the quality of the data, which is usually difficult in practice, 
the mere presentation of the data in the right format may be no more than 'window 
dressing’ to satisfy the demands of the agencies concerned.” (Hill, 1999, p.1) 


Implementation by NSOs of international standards like the SNA 1993 tends to be 
selective and judgmental for the following pragmatic reasons. 


Some parts of the SNA 1993 do not have the general support of economists and 
national accountants. Some illustrations involve attempts at early updates to the SNA 
1993 which were later reversed. For example, the expert group’s decision to capitalise 
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Research and Development expenditures was subsequently reversed because of 
objections by many NSOs. Another example is of some NSOs now thinking the SNA 
1993 has not gone far enough in the area of regarding all defence durables which can 
equally well be used for civilian purposes be treated as assets rather than 
consumables. 


Some international standards are not always consistent internally. In the SNA 1993 the 
treatment of intangibles and royalties is an example. (Hill, 1999, p. 2) Another is the 
conceptually flawed SNA 1993 definition of global output regarding financial 
institutions services indirectly measured (FISIM) and definition of the financial sector. 


“The expert group on the revision of the SNA abandoned the attempt to find general 
definitions for ‘money’ and ‘banks’ which would be both meaningful and useful across a 
broad spectrum of countries.” (Hill, 1999, p. 6) 


This is an area of concern because financial institutions and financial instruments are 
continually evolving into new forms, and play a greater role in the new economy. 
(Hill, 1999, p. 7) Also, the treatment of the interest on bonds in SNA 1993 and BPM 5 
is not always satisfactory. By recording the gains but not the consequential interest 
changes the system of accounts become internally inconsistent. (Hill, 1999, p. 8) 


Some areas of the SNA 1993 are ambiguous, so that judgement by NSOs has to be 
exercised. The allocation or non-allocation of the output of financial institutions 
services indirectly measured (FISIM) was not resolved in SNA 1993. Depending on the 
economic significance of the financial sector to economic activity the allocation or 
otherwise of output from FISIM can have a noticeable impact on GDP growth. In this 
area the ABS has taken a leadership role: 


“The method of implementing FISIM developed by ABS is likely to form the basis for an 
updated international standard in this area.” (Hill, 1999, p. 7) 


In some areas the SNA is clear, direct and unambiguous about what should be done, 
but is not explicit about how it should be achieved. This raises particular dilemmas for 
NSOs and for quality assessments of SEAs. For example, the treatment of illegal and 
underground economic activities, the so called black economy. SNA 1993 requires 
most illegal activities and transactions to be recorded in the accounts. This is because 
logically the legal and illegal economies are interdependent and interact with each 
other, and consequently constitute a single economic system. To exclude the black 
economy from the accounts would introduce errors into the key aggregates of the 
accounts, and render the system internally inconsistent and misleading for analysis. 
However, the practical difficulty of obtaining any reliable data regarding illegal 
activities and transactions is self evident. But because of the nature of these particular 
activities SNA 1993 cannot provide guidance to countries as to how in practice the 
economic value of these activities can be best estimated. 
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In the ABS nine methods have been considered for estimation of the black economy, 
but which method is appropriate is conjectural. This area raises particular problems 
for quality assessments. Internationally this area tends to be neglected because of the 
sensitivities involved and practical methodological difficulties, and the quality of the 
systems of economic accounts suffer accordingly. 
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